Search Results: "alexander"

18 April 2016

Reproducible builds folks: Reproducible builds: week 50 in Stretch cycle

What happened in the reproducible builds effort between April 3rd and April 9th 2016: Media coverage Emily Ratliff wrote an article for SecurityWeek called Establishing Correspondence Between an Application and its Source Code - How Combining Two Completely Separate Open Source Projects Can Make Us All More Secure. Tails have started work on a design for freezable APT repositories to make it easier and practical to perform reproductions of an entire distribution at a given point in time, which will be needed to create reproducible installation- or live-media. Toolchain fixes Alexis Bienven e submitted patches adding support for SOURCE_DATE_EPOCH in several tools: transfig, imagemagick, rdtool, and asciidoctor. boyska submitted one for python-reportlab. Packages fixed The following packages have become reproducible due to changes in their build dependencies: atinject-jsr330 brailleutils cglib3 gnugo libcobra-java libgnumail-java libjchart2d-java libjcommon-java libjfreechart-java libjide-oss-java liblaf-widget-java liblastfm-java liboptions-java octave-control octave-mpi octave-nan octave-parallel octave-stk octave-struct octave-tsa oar The following packages became reproducible after getting fixed: Several uploads fixed some reproducibility issues, but not all of them: Patches submitted which have not made their way to the archive yet: Other upstream fixes Alexander Batischev made a commit to make newsbeuter reproducible. tests.reproducible-builds.org Package reviews 93 reviews have been removed, 66 added and 21 updated in the previous week. 12 new FTBFS bugs have been reported by Chris Lamb and Niko Tyni. Misc. This week's edition was written by Lunar, Holger Levsen, Reiner Herrmann, Mattia Rizzolo and Ximin Luo. With the departure of Lunar as a full-time contributor, Reproducible Builds Weekly News (this thing you're reading) has moved from his personal Debian blog on Debian People to the Reproducible Builds team web site on Debian Alioth. You may want to update your RSS or Atom feeds. Very many thanks to Lunar for writing and publishing this weekly news for so long, well & continously!

1 February 2016

Lunar: Reproducible builds: week 40 in Stretch cycle

What happened in the reproducible builds effort between January 24th and January 30th:

Media coverage Holger Levsen was interviewed by the FOSDEM team to introduce his talk on Sunday 31st.

Toolchain fixes Jonas Smedegaard uploaded d-shlibs/0.63 which makes the order of dependencies generated by d-devlibdeps stable accross locales. Original patch by Reiner Herrmann.

Packages fixed The following 53 packages have become reproducible due to changes in their build dependencies: appstream-glib, aptitude, arbtt, btrfs-tools, cinnamon-settings-daemon, cppcheck, debian-security-support, easytag, gitit, gnash, gnome-control-center, gnome-keyring, gnome-shell, gnome-software, graphite2, gtk+2.0, gupnp, gvfs, gyp, hgview, htmlcxx, i3status, imms, irker, jmapviewer, katarakt, kmod, lastpass-cli, libaccounts-glib, libam7xxx, libldm, libopenobex, libsecret, linthesia, mate-session-manager, mpris-remote, network-manager, paprefs, php-opencloud, pisa, pyacidobasic, python-pymzml, python-pyscss, qtquick1-opensource-src, rdkit, ruby-rails-html-sanitizer, shellex, slony1-2, spacezero, spamprobe, sugar-toolkit-gtk3, tachyon, tgt. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues, but not all of them:
  • gnubg/1.05.000-4 by Russ Allbery.
  • grcompiler/4.2-6 by Hideki Yamane.
  • sdlgfx/2.0.25-5 fix by Felix Geyer, uploaded by Gianfranco Costamagna.
Patches submitted which have not made their way to the archive yet:
  • #812876 on glib2.0 by Lunar: ensure that functions are sorted using the C locale when giotypefuncs.c is generated.

diffoscope development diffoscope 48 was released on January 26th. It fixes several issues introduced by the retrieval of extra symbols from Debian debug packages. It also restores compatibility with older versions of binutils which does not support readelf --decompress.

strip-nondeterminism development strip-nondeterminism 0.015-1 was uploaded on January 27th. It fixes handling of signed JAR files which are now going to be ignored to keep the signatures intact.

Package reviews 54 reviews have been removed, 36 added and 17 updated in the previous week. 30 new FTBFS bugs have been submitted by Chris Lamb, Michael Tautschnig, Mattia Rizzolo, Tobias Frost.

Misc. Alexander Couzens and Bryan Newbold have been busy fixing more issues in OpenWrt. Version 1.6.3 of FreeBSD's package manager pkg(8) now supports SOURCE_DATE_EPOCH. Ross Karchner did a lightning talk about reproducible builds at his work place and shared the slides.

27 December 2015

Vincent Sanders: The only pleasure I get from moving house is stumbling across books I had forgotton I owned

I have to agree with John Burnside on that statement, after having recently moved house again rediscovering our book collection has been a salve for an otherwise exhausting undertaking. I returned to Cambridge four years ago, initially on my own and then subsequently the family moved down to be with me.

We rented a house but, with two growing teenagers, the accommodation was becoming a little crowded. Melodie and I decided the relocation was permanent and started looking for our own property, eventually finding something to our liking in Cottenham village.

Melodie took the opportunity to have the house cleaned and decorated while empty because of overlapping time with our rental property. This meant we had to be a little careful while moving in as there was still wet paint in places.

Some of our books
Moving weekend was made bearable by Steve, Jonathan and Jo lending a hand especially on the trips to Yorkshire to retrieve, amongst other things, the aforementioned book collection. We were also fortunate to have Andy and Jane doing many other important jobs around the place while the rest of us were messing about in vans.

The desk in the study
The seemingly obligatory trip to IKEA to acquire furniture was made much more fun by trying to park a luton van which was only possible because Steve and Jonathan helped me. Though it turns out IKEA ship mattresses rolled up so tight they can be moved in an estate car so taking the van was unnecessary.

Alex under his loft bed
Having moved in it seems like every weekend is filled with a never ending "todo" list of jobs. From clearing gutters to building a desk in the study. Eight weeks on and the list seems to be slowly shrinking meaning I can even do some lower priority things like the server rack which was actually a fun project.


Joshua in his completed roomThe holidays this year afforded me some time to finish the boys bedrooms. They both got loft beds with a substantial area underneath. This allows them both to have double beds along with a desk and plenty of storage. Completing the rooms required the construction of some flat pack furniture which rather than simply do myself I supervised the boys doing it themselves.

Alexander building flat pack furniture
Teaching them by letting them get on with it was a surprisingly effective and both of them got the hang of the construction method pretty quickly. There was only a couple of errors from which they learned immediately and did not repeat (draw bottoms having a finished side and front becomes back when you are constructing upside down)

Joshua assembling flat pack furniture
The house is starting to feel like home and soon all the problems will fade from memory while the good will remain. Certainly our first holiday season has been comfortable here and I look forward to many more re-reading our books.

23 December 2015

Alexander Wirt: New mailinglists

Today I spend some time to go through the lists.debian.org bugreports. In consequence I created three new lists:

11 December 2015

Lunar: Reproducible builds: week 32 in Stretch cycle

The first reproducible world summit was held in Athens, Greece, from December 1st-3rd with the support of the Linux Foundation, the Open Tech Fund, and Google. Faidon Liambotis has been an amazing help to sort out all local details. People at ImpactHub Athens have been perfect hosts. North of Athens from the Acropolis with ImpactHub in the center Nearly 40 participants from 14 different free software project had very busy days sharing knowledge, building understanding, and producing actual patches. Anyone interested in cross project discussions should join the rb-general mailing-list. What follows focuses mostly on what happened for Debian this previous week. A more detailed report about the summit will follow soon. You can also read the ones from Joachim Breitner from Debian, Clemens Lang from MacPorts, Georg Koppen from Tor, Dhiru Kholia from Fedora, and Ludovic Court s wrote one for Guix and for the GNU project. The Acropolis from Infrastructure Several discussions at the meeting helped refine a shared understanding of what kind of information should be recorded on a build, and how they could be used. Daniel Kahn Gillmor sent a detailed update on how .buildinfo files should become part of the Debian archive. Some key changes compared to what we had in mind at DebConf15: Hopefully, ftpmasters will be able to comment on the updated proposal soon. Packages fixed The following packages have become reproducible due to changes in their build dependencies: fades, triplane, caml-crush, globus-authz. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues, but not all of them: Patches submitted which have not made their way to the archive yet: akira sent proposals on how to make bash reproducible. Alexander Couzens submitted a patch upstream to add support for SOURCE_DATE_EPOCH in grub image generator (#787795). reproducible.debian.net An issue with some armhf build nodes was tracked down to a bad interaction between uname26 personality and new glibc (Vagrant Cascadian). A Debian package was created for koji, the RPM building and tracking system used by Fedora amongst others. It is currently waiting for review in the NEW queue. (Ximin Luo, Marek Marczykowski-G recki) diffoscope development diffoscope now has a dedicated mailing list to better accommodate its growing user and developer base. Going through diffoscope's guts together enabled several new contributors. Baptiste Daroussin, Ed Maste, Clemens Lang, Mike McQuaid, Joachim Breitner all contributed their first patches to improve portability or add new features. Regular contributors Chris Lamb, Reiner Herrmann, and Levente Polyak also submitted improvements. diffoscope hacking session in Athens The next release should support more operating systems, filesystem image comparison via libguestfs, HTML reports with on-demand loading, and parallel processing for the most noticeable improvements. Package reviews 27 reviews have been removed, 17 added and 14 updated in the previous week. Chris Lamb and Val Lorentz filed 4 new FTBFS reports. Misc. Baptiste Daroussin has started to implement support for SOURCE_DATE_EPOCH in FreeBSD in libpkg and the ports tree. Thanks Joachim Breitner and h01ger for the pictures.

9 November 2015

Daniel Pocock: debian.org RTC: announcing XMPP, SIP presence and more

Announced 7 November 2015 on the debian-devel-announce mailing list. The Debian Project now has an XMPP service available to all Debian Developers. Your Debian.org email identity can be used as your XMPP address. The SIP service has also been upgraded and now supports presence. SIP and XMPP presence, rosters and messaging are not currently integrated. The Lumicall app has been improved to enable rapid setup for Debian.org SIP users. This announcement concludes the maintenance window on the RTC services. All services are now running on jessie (using packages from jessie-backports). XMPP and SIP enable a whole new world of real-time multimedia communications possibilities: video/webcam, VoIP, chat messaging, desktop sharing and distributed, federated communication are the most common use cases. Details about how to get started and get support are explained in the User Guide in the Debian wiki. As it is a wiki, you are completely welcome to help it evolve. Several of the people involved in the RTC team were also at the Cambridge mini-DebConf (7-8 November). The password for all these real time communication services can be set via the LDAP control panel. Please note that this password needs to be different to any of your other existing debian.org passwords. Please use a strong password and please keep it secure. Some of the infrastructure, like the TURN server, is shared by clients of both SIP and XMPP. Please configure your XMPP and SIP clients to use the TURN server for audio or video streaming to work most reliably through NAT. A key feature of both our XMPP and SIP services is that they support federated inter-connectivity with other domains. Please try it. The FedRTC service for Fedora developers is one example of another SIP service that supports federation. For details of how it works and how we establish trust between domains, please see the RTC Quick Start Guide. Please reach out to other communities you are involved with and help them consider enabling SIP and XMPP federation of their own communities/domains: as Metcalfe's law suggests, each extra person or community who embraces open standards like SIP and XMPP has far more than just an incremental impact on the value of these standards and makes them more pervasive. If you are keen to support and collaborate on the wider use of Free RTC technology, please consider joining the Free RTC mailing list sponsored by FSF Europe. There will also be a dedicated debian-rtc list for discussion of these technologies within Debian and derivatives. This service has been made possible by the efforts of the DSA team in the original SIP+WebRTC project and the more recent jessie upgrades and XMPP project. Real-time communications systems have specific expectations for network latency, connectivity, authentication schemes and various other things. Therefore, it is a great endorsement of the caliber of the team and the quality of the systems they have in place that they have been able to host this largely within their existing framework for Debian services. Feedback from the DSA team has also been helpful in improving the upstream software and packaging to make them convenient for system administrators everywhere. Special thanks to Peter Palfrader and Luca Filipozzi from the DSA team, Matthew Wild from the Prosody XMPP server project, Scott Godin from the reSIProcate project, Juliana Louback for her contributions to JSCommunicator during GSoC 2014, Iain Learmonth for helping get the RTC team up and running, Enrico Tassi, Sergei Golovan and Victor Seva for the Prosody and prosody-modules packaging and also the Debian backports team, especially Alexander Wirt, helping us ensure that rapidly evolving packages like those used in RTC are available on a stable Debian system.

17 September 2015

Lunar: A key signing party keyserver as a Tor hidden service

Key signing parties are a pain and hopefully, one day, we will have better ways to authentication keys than reading hexadecimal strings out loud. The Zimmermann Sassaman key-signing protocol makes them much more bearable already by having only one single hexadecimal string read out loud. That string is the cryptographic hash of a document given to every participant listing all participants and their fingerprints. If everyone has the same hash, then we assume that everyone has the same document. Then, participants in turn will confirm that they fully recognize the fingerprint listed in the document. Alexander Wirt wrote a small key server dedicated to receive keys from the participants. There is also a script that will generate the document from the submitted keys and a ready-to-use keyring. The latter can be run automatically using inoticoming when a new key arrives. Finally, it would be nice if participants could confirm that their key has been properly added to the document, e.g. by making the list available on a web server. Setting all this up seemed like a good opportunity to play with Tor hidden services and systemd-nspawn. Here's the setup log with some comments. This was done on a small armhf device with Debian Jessie. Create a new hidden service Edit /etc/tor/torrc on the host to setup the hidden service: 
HiddenServiceDir /var/lib/tor/ksp/
HiddenServicePort 80 10.0.0.2:80
HiddenServicePort 11371 10.0.0.2:11371
Run: 
host# systemctl reload tor.service
Then, to learn the name of the newly created hidden service name: 
host# cat /var/lib/tor/ksp/hostname
ksp123456789abcd.onion
Install the container debootstrap as always: 
host# debootstrap --variant=minbase jessie /var/lib/container/ksp
Preliminary container configuration We do the following step simply using chroot as we are going to use the host network configuration for this stage. The container itself will not have access to the Internet. 
host# chroot ksp
Let's set the hostname: 
ksp-chroot# echo 'ksp' > /etc/hostname
Set up APT: 
ksp-chroot# echo 'deb http://httpredir.debian.org/debian jessie main' > /etc/apt/sources.list
ksp-chroot# apt update
We need dbus to get systemd to work well: 
ksp-chroot# apt-get install dbus
Make sure that we can resolve our own hostname: 
ksp-chroot# apt-get install libnss-myhostname
ksp-chroot# sed -e '/^hosts:/s/files/myhostname \0/' -i /etc/nsswitch.conf
These are dependencies of the keyserver: 
ksp-chroot# apt-get install --no-install-recommends libhttp-daemon-perl \
                liblog-loglite-perl libproc-reliable-perl
These ones are needed for the script generating the list: 
ksp-chroot# apt-get install bzip2 inoticoming
And we will use the smallest HTTP server available: 
ksp-chroot# apt-get install netcat-traditional micro-httpd
Finally, let's unconfigure all DNS resolvers: 
ksp-chroot# echo > /etc/resolv.conf
And we are done with the chroot: 
ksp-chroot# exit
Let's retrieve the ksp-tools repository now: 
host# cd /var/lib/container/srv
host# git clone https://github.com/formorer/ksp-tools
Container setup We will now start the container with a shell to configure it: 
host# systemd-nspawn -D ksp --network-veth
Let's ask systemd to configure the network for us: 
ksp# systemctl enable systemd-networkd
Let's not forget to set a root password: 
ksp# passwd
We add a dedicated user to run the keyserver and the list generation script: 
ksp# adduser --system --group --disabled-password --disabled-login --home /var/lib/ksp ksp
Let's configure the keyserver: 
ksp# cp /srv/ksp-tools/keyserver.conf /var/lib/ksp/keyserver.conf
Let's edit /var/lib/ksp/keyserver.conf: 
homedir = /var/lib/ksp
Now create the GnuPG homedir for the keyserve: 
ksp# mkdir /var/lib/ksp/keys
ksp# install -d -o ksp -g ksp -m 0700 /var/lib/ksp/keys/gpg
Copy the template list generator: 
ksp# cp -r /srv/ksp-tools/example /var/lib/ksp/keys/ksp123456789abcd_onion
Create the key repository: 
ksp# install -d -o ksp -g ksp -m 0700 /var/lib/ksp/keys/ksp123456789abcd_onion/keys
Create a directory accessible to the web server where the participant list will be generated: 
ksp# mkdir -p /var/www
ksp# install -d -o ksp -g ksp -m 0755 /var/www/keys
Let's configure the list generation script by editing /var/lib/ksp/keys/ksp123456789abcd_onion/conf/vars: 
KS=ksp123456789abcd.onion
export GNUPGHOME=/tmp/ksp-gpg
KSPFILE="/var/www/keys/ksp-event.txt"
Don't forget to adjust the header in /var/lib/ksp/keys/ksp123456789abcd_onion/conf/list-header. Now we create a unit file for the keyserver in /etc/systemd/system/keyserver.service: 
[Unit]
Description=Key signing party keyserver
[Service]
Type=simple
Environment="KSP_HOMEDIR=/var/lib/ksp"
ExecStart=/srv/ksp-tools/bin/kspkeyserver.pl --nodaemonize
User=ksp
Group=ksp
PrivateTmp=yes
ProtectHome=yes
ProtectSystem=full
ReadOnlyDirectories=/
ReadWriteDirectories=-/var/lib/ksp
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
[Install]
WantedBy=multi-user.target
Another unit for the list generator as /etc/systemd/system/ksp-list-generator.service: 
[Unit]
Description=Key signing party list generator
[Service]
Type=simple
EnvironmentFile=/var/lib/ksp/keys/ksp123456789abcd_onion/conf/vars
ExecStart=/usr/bin/inoticoming --foreground /var/lib/ksp/keys/ksp123456789abcd_onion/keys --chdir /var/lib/ksp/keys/ksp123456789abcd_onion bin/generate-list \;
User=ksp
Group=ksp
PrivateTmp=yes
ProtectHome=yes
ProtectSystem=full
ReadOnlyDirectories=/
ReadWriteDirectories=/var/www/keys
CapabilityBoundingSet=
[Install]
WantedBy=multi-user.target
For the web server, we first configure a socket listening on port 80 in /etc/systemd/system/micro-httpd.socket: 
[Unit]
Description=micro-httpd socket
[Socket]
ListenStream=80
Accept=yes
[Install]
WantedBy=sockets.target
And then the web server in /etc/systemd/system/micro-httpd@.service: 
[Unit]
Description=micro-httpd server
[Service]
ExecStart=-/usr/sbin/micro-httpd /var/www/ksp
StandardInput=socket
PrivateTmp=yes
ProtectHome=yes
ProtectSystem=full
ReadOnlyDirectories=/
CapabilityBoundingSet=
Let's now ask systemd to start all of these at boot time: 
ksp# systemctl daemon-reload
ksp# systemctl enable keyserver.service
ksp# systemctl enable ksp-list-generator.service
ksp# systemctl enable micro-httpd.socket
One way to kill the container is to type Control+] three times. Boot the container Let's get this party started! 
host# systemd-nspawn -b -D /var/lib/container/ksp --network-veth
Hopefully, things should work now. Participants to the KSP should then be able to send their key with: 
$ torsocks gpg --keyserver ksp123456789abcd.onion --send-key $KEYID
(Sadly, this is broken with GnuPG 2.1 at the moment.) The participant list should be available at http://ksp123456789abcd.onion/ksp-event.txt. Final steps We need to tell systemd to start the container started at boot time: 
host# systemctl enable systemd-nspawn@ksp.service
But the default command-line will not use a dedicated network, so we need to override that part of the configuration. First create a directory: 
host# mkdir /etc/systemd/system/systemd-nspawn@ksp.service.d
And edit /etc/systemd/system/systemd-nspawn@ksp.service.d/use-network-veth.conf: 
[Service]
# The empty line because we want to override all previous ExecStart
# and not add an extra command
ExecStart=
ExecStart=/usr/bin/systemd-nspawn --quiet --keep-unit --boot --link-journal=try-guest --directory=/var/lib/container/%i --network-veth
Let's reload systemd and verify that our snippet is there: 
host# systemctl daemon-reload
host# systemctl cat systemd-nspawn@ksp.service
All good? Let's start it: 
host# systemctl start systemd-nspawn@ksp.service
One should also add a firewall to disallow any outgoing connections from the ve-ksp interface as an extra protection.

13 September 2015

Gregor Herrmann: RC bugs 2015/31-37

during the last weeks, I spent time mostly with RC bug prevention but I at least managed to also fix a couple of actual RC bugs:

27 August 2015

Alexander Wirt: Basic support for SSO Client certificates on paste.debian.net

Sometimes waiting for a delayed flight helps to implement things. I added some basic support for the new Debian SSO Client Certificate feature to paste.debian.net. If you are using such a certificate most anti-spam restrictions, code limitations and so on won t count for you anymore.

19 July 2015

Gregor Herrmann: RC bugs 2015/17-29

after the release is before the release. or: long time no RC bug report. after the jessie release I spent most of my Debian time on work in the Debian Perl Group. we tried to get down the list of new upstream releases (from over 500 to currently 379; unfortunately the CPAN never sleeps), we were & still are busy preparing for the Perl 5.22 transition (e.g. we uploaded something between 300 & 400 packages to deal with Module::Build & CGI.pm being removed from perl core; only team-maintained packages so far), & we had a pleasant & productive sprint in Barcelona in May. & I also tried to fix some of the RC bugs in our packages which popped up over the previous months. yesterday & today I finally found some time to help with the GCC 5 transition, mostly by making QA or Non-Maintainer Uploads with patches that already were in the BTS. a big thanks especially to the team at HP which provided a couple dozens patches! & here's the list of RC bugs I've worked on in the last 3 months:

20 June 2015

Lunar: Reproducible builds: week 5 in Stretch cycle

What happened about the reproducible builds effort for this week: Toolchain fixes Uploads that should help other packages: Patch submitted for toolchain issues: Some discussions have been started in Debian and with upstream: Packages fixed The following 8 packages became reproducible due to changes in their build dependencies: access-modifier-checker, apache-log4j2, jenkins-xstream, libsdl-perl, maven-shared-incremental, ruby-pygments.rb, ruby-wikicloth, uimaj. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: Patches submitted which did not make their way to the archive yet: Discussions that have been started: reproducible.debian.net Holger Levsen added two new package sets: pkg-javascript-devel and pkg-php-pear. The list of packages with and without notes are now sorted by age of the latest build. Mattia Rizzolo added support for email notifications so that maintainers can be warned when a package becomes unreproducible. Please ask Mattia or Holger or in the #debian-reproducible IRC channel if you want to be notified for your packages! strip-nondeterminism development Andrew Ayer fixed the gzip handler so that it skip adding a predetermined timestamp when there was none. Documentation update Lunar added documentation about mtimes of file extracted using unzip being timezone dependent. He also wrote a short example on how to test reproducibility. Stephen Kitt updated the documentation about timestamps in PE binaries. Documentation and scripts to perform weekly reports were published by Lunar. Package reviews 50 obsolete reviews have been removed, 51 added and 29 updated this week. Thanks Chris West and Mathieu Bridon amongst others. New identified issues: Misc. Lunar will be talking (in French) about reproducible builds at Pas Sage en Seine on June 19th, at 15:00 in Paris. Meeting will happen this Wednesday, 19:00 UTC.

8 June 2015

Lunar: Reproducible builds: week 6 in Stretch cycle

What happened about the reproducible builds effort for this week: Presentations On May 26th,Holger Levsen presented reproducible builds in Debian at CCC Berlin for the Datengarten 52. The presentation was in German and the slides in English. Audio and video recordings are available. Toolchain fixes Niels Thykier fixed the experimental support for the automatic creation of debug packages in debhelper that being tested as part of the reproducible toolchain. Lunar added to the reproducible build version of dpkg the normalization of permissions for files in control.tar. The patch has also been submitted based on the main branch. Daniel Kahn Gillmor proposed a patch to add support for externally-supplying build date to help2man. This sparkled a discussion about agreeing on a common name for an environment variable to hold the date that should be used. It seems opinions are converging on using SOURCE_DATE_UTC which would hold a ISO-8601 formatted date in UTC) (e.g. 2015-06-05T01:08:20Z). Kudos to Daniel, Brendan O'Dea, Ximin Luo for pushing this forward. Lunar proposed a patch to Tar upstream adding a --clamp-mtime option as a generic solution for timestamp variations in tarballs which might also be useful for dpkg. The option changes the behavior of --mtime to only use the time specified if the file mtime is newer than the given time. So far, upstream is not convinced that it would make a worthwhile addition to Tar, though. Daniel Kahn Gillmor reached out to the libburnia project to ask for help on how to make ISO created with xorriso reproducible. We should reward Thomas Schmitt with a model upstream trophy as he went through a thorough analysis of possible sources of variations and ways to improve the situation. Most of what is missing with the current version in Debian is available in the latest upstream version, but libisoburn in Debian needs help. Daniel backported the missing option for version 1.3.2-1.1. akira submitted a new issue to Doxygen upstream regarding the timestamps added to the generated manpages. Packages fixed The following 49 packages became reproducible due to changes in their build dependencies: activemq-protobuf, bnfc, bridge-method-injector, commons-exec, console-data, djinn, github-backup, haskell-authenticate-oauth, haskell-authenticate, haskell-blaze-builder, haskell-blaze-textual, haskell-bloomfilter, haskell-brainfuck, haskell-hspec-discover, haskell-pretty-show, haskell-unlambda, haskell-x509-util, haskelldb-hdbc-odbc, haskelldb-hdbc-postgresql, haskelldb-hdbc-sqlite3, hasktags, hedgewars, hscolour, https-everywhere, java-comment-preprocessor, jffi, jgit, jnr-ffi, jnr-netdb, jsoup, lhs2tex, libcolor-calc-perl, libfile-changenotify-perl, libpdl-io-hdf5-perl, libsvn-notify-mirror-perl, localizer, maven-enforcer, pyotherside, python-xlrd, python-xstatic-angular-bootstrap, rt-extension-calendar, ruby-builder, ruby-em-hiredis, ruby-redcloth, shellcheck, sisu-plexus, tomcat-maven-plugin, v4l2loopback, vim-latexsuite. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: Patches submitted which did not make their way to the archive yet: Daniel Kahn Gilmor also started discussions for emacs24 and the unsorted lists in generated .el files, the recording of a PID number in lush, and the reproducibility of ISO images in grub2. reproducible.debian.net Notifications are now sent when the build environment for a package has changed between two builds. This is a first step before automatically building the package once more. (Holger Levsen) jenkins.debian.net was upgraded to Debian Jessie. (Holger Levsen) A new variation is now being tested: $PATH. The second build will be done with a /i/capture/the/path added. (Holger Levsen) Holger Levsen with the help of Alexander Couzens wrote extra job to test the reproducibility of coreboot. Thanks James McCoy for helping with certificate issues. Mattia Rizollo made some more internal improvements. strip-nondeterminism development Andrew Ayer released strip-nondeterminism/0.008-1. This new version fixes the gzip handler so that it now skip adding a predetermined timestamp when there was none. Holger Levsen sponsored the upload. Documentation update The pages about timestamps in manpages generated by Doxygen, GHC .hi files, and Jar files have been updated to reflect their status in upstream. Markus Koschany documented an easy way to prevent Doxygen to write timestamps in HTML output. Package reviews 83 obsolete reviews have been removed, 71 added and 48 updated this week. Meetings A meeting was held on 2015-06-03. Minutes and full logs are available. It was agreed to hold such a meeting every two weeks for the time being. The time of the next meeting should be announced soon.

31 March 2015

Dirk Eddelbuettel: R / Finance 2015 Open for Registration

The annoucement below just went to the R-SIG-Finance list. More information is as usual at the R / Finance page.
Registration for R/Finance 2015 is now open! The conference will take place on May 29 and 30, at UIC in Chicago. Building on the success of the previous conferences in 2009-2014, we expect more than 250 attendees from around the world. R users from industry, academia, and government will joining 30+ presenters covering all areas of finance with R. We are very excited about the four keynote presentations given by Emanuel Derman, Louis Marascio, Alexander McNeil, and Rishi Narang.
The conference agenda (currently) includes 18 full presentations and 19 shorter "lightning talks". As in previous years, several (optional) pre-conference seminars are offered on Friday morning. There is also an (optional) conference dinner at The Terrace at Trump Hotel. Overlooking the Chicago river and skyline, it is a perfect venue to continue conversations while dining and drinking. Registration information and agenda details can be found on the conference website as they are being finalized.
Registration is also available directly at the registration page. We would to thank our 2015 sponsors for the continued support enabling us to host such an exciting conference: International Center for Futures and Derivatives at UIC Revolution Analytics
MS-Computational Finance and Risk Management at University of Washington Ketchum Trading
OneMarketData
RStudio
SYMMS On behalf of the committee and sponsors, we look forward to seeing you in Chicago! For the program committee:
Gib Bassett, Peter Carl, Dirk Eddelbuettel, Brian Peterson,
Dale Rosenthal, Jeffrey Ryan, Joshua Ulrich
See you in Chicago in May!

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

30 March 2015

Matthias Klumpp: Limba Project: Another progress report

And once again, it s time for another Limba blogpost :-)limba-small Limba is a solution to install 3rd-party software on Linux, without interfering with the distribution s native package manager. It can be useful to try out different software versions, use newer software on a stable OS release or simply to obtain software which does not yet exist for your distribution. Limba works distribution-independent, so software authors only need to publish their software once for all Linux distributions. I recently released version 0.4, with which all most important features you would expect from a software manager are complete. This includes installing & removing packages, GPG-signing of packages, package repositories, package updates etc. Using Limba is still a bit rough, but most things work pretty well already. So, it s time for another progress report. Since a FAQ-like list is easier to digest. compared to a long blogpost, I go with this again. So, let s address one important general question first: How does Limba relate to the GNOME Sandboxing approach? (If you don t know about GNOMEs sandboxes, take a look at the GNOME Wiki Alexander Larsson also blogged about it recently) First of all: There is no rivalry here and no NIH syndrome involved. Limba and GNOMEs Sandboxes (XdgApp) are different concepts, which both have their place. The main difference between both projects is the handling of runtimes. A runtime is the shared libraries and other shared ressources applications use. This includes libraries like GTK+/Qt5/SDL/libpulse etc. XdgApp applications have one big runtime they can use, built with OSTree. This runtime is static and will not change, it will only receive critical security updates. A runtime in XdgApp is provided by a vendor like GNOME as a compilation of multiple single libraries. Limba, on the other hand, generates runtimes on the target system on-the-fly out of several subcomponents with dependency-relations between them. Each component can be updated independently, as long as the dependencies are satisfied. The individual components are intended to be provided by the respective upstream projects. Both projects have their individual up and downsides: While the static runtime of XdgApp projects makes testing simple, it is also harder to extend and more difficult to update. If something you need is not provided by the mega-runtime, you will have to provide it by yourself (e.g. we will have some applications ship smaller shared libraries with their binaries, as they are not part of the big runtime). Limba does not have this issue, but instead, with its dynamic runtimes, relies on upstreams behaving nice and not breaking ABIs in security updates, so existing applications continue to be working even with newer software components. Obviously, I like the Limba approach more, since it is incredibly flexible, and even allows to mimic the behaviour of GNOMEs XdgApp by using absolute dependencies on components. Do you have an example of a Limba-distributed application? Yes! I recently created a set of package for Neverball Alexander Larsson also created a XdgApp bundle for it, and due to the low amount of stuff Neverball depends on, it was a perfect test subject. One of the main things I want to achieve with Limba is to integrate it well with continuous integration systems, so you can automatically get a Limba package built for your application and have it tested with the current set of dependencies. Also, building packages should be very easy, and as failsafe as possible. You can find the current Neverball test in the Limba-Neverball repository on Github. All you need (after installing Limba and the build dependencies of all components) is to run the make_all.sh script. Later, I also want to provide helper tools to automatically build the software in a chroot environment, and to allow building against the exact version depended on in the Limba package. Creating a Limba package is trivial, it boils down to creating a simple control file describing the dependencies of the package, and to write an AppStream metadata file. If you feel adventurous, you can also add automatic build instructions as a YAML file (which uses a subset of the Travis build config schema) This is the Neverball Limba package, built on Tanglu 3, run on Fedora 21: Limba-installed Neverball Which kernel do I need to run Limba? The Limba build tools run on any Linux version, but to run applications installed with Limba, you need at least Linux 3.18 (for Limba 0.4.2). I plan to bump the minimum version requirement to Linux 4.0+ very soon, since this release contains some improvements in OverlayFS and a few other kernel features I am thinking about making use of. Linux 3.18 is included in most Linux distributions released in 2015 (and of course any rolling release distribution and Fedora have it). Building all these little Limba packages and keeping them up-to-date is annoying Yes indeed. I expect that we will see some bigger Limba packages bundling a few dependencies, but in general this is a pretty annoying property of Limba currently, since there are so few packages available you can reuse. But I plan to address this. Behind the scenes, I am working on a webservice, which will allow developers to upload Limba packages. This central ressource can then be used by other developers to obtain dependencies. We can also perform some QA on the received packages, map the available software with CVE databases to see if a component is vulnerable and publish that information, etc. All of this is currently planned, and I can t say a lot more yet. Stay tuned! (As always: If you want to help, please contact me) Are the Limba interfaces stable? Can I use it already? The Limba package format should be stable by now since Limba is still Alpha software, I will however, make breaking changes in case there is a huge flaw which makes it reasonable to break the IPK package format. I don t think that this will happen though, as the Limba packages are designed to be easily backward- and forward compatible. For the Limba repository format, I might make some more changes though (less invasive, but you might need to rebuilt the repository). tl;dr: Yes! Plase use Limba and report bugs, but keep in mind that Limba is still in an early stage of development, and we need bug reports! Will there be integration into GNOME-Software and Muon? From the GNOME-Software side, there were positive signals about that, but some technical obstancles need to be resolved first. I did not yet get in contact with the Muon crew they are just implementing AppStream, which is a prerequisite for having any support for Limba[1]. Since PackageKit dropped the support for plugins, every software manager needs to implement support for Limba.
So, thanks for reading this (again too long) blogpost :) There are some more exciting things coming soon, especially regarding AppStream on Debian/Ubuntu! [1]: And I should actually help with the AppStream support, but currently I can not allocate enough time to take that additional project as well this might change in a few weeks. Also, Muon does pretty well already!

16 March 2015

Russ Allbery: Another haul post

Wow, it's been quite a long time since I've posted something here. Everything is going well -- I'm just very, very engrossed with the new job, since I'm still in exponential ramp-up mode. It's lasting for longer than I expected, although my expectations didn't have much basis since this is the first time I've started a new job in 17 years. I'm feeling more and more capable every day, but the combination of a very heavily social learning process, a lot of new technical areas to learn, and not having taken a vacation since last June means that my weekends are spent just passively watching things and zoning. Not sure yet how long that will last, and I don't want to make any predictions, although I do have my first significant vacation coming up next month. Anyway, book reading and buying has continued, although I'm again far behind on writing reviews. With luck, I'll be writing one of those (for posting later) right after writing this post. Michelle Alexander The New Jim Crow (non-fiction)
Elizabeth Bear Karen Memory (sff)
Becky Chambers The Long Way to a Small, Angry Planet (sff)
Fred Clark The Anti-Christ Handbook (non-fiction)
Charles de Lint The Very Best of Charles de Lint (sff)
S.L. Huang A Neurological Study on the Effects... (sff)
S.L. Huang Half Life (sff)
Kameron Hurley The Mirror Empire (sff)
Sophie Lack Dissonance (sff)
Sophie Lack Imbalance (sff)
Susan R. Matthews An Exchange of Hostages (sff)
Kaoru Mori A Bride's Story #1 (graphic novel)
Donald Shoup The High Cost of Free Parking (non-fiction)
Jo Walton The Just City (sff) Pretty nice variety of different stuff from a huge variety of recommendation sources. I've already read the Chambers (and can recommend it). A review will be forthcoming.

22 December 2014

Michael Prokop: Ten years of Grml

* On 22nd of October 2004 an event called OS04 took place in Seifenfabrik Graz/Austria and it marked the first official release of the Grml project. Grml was initially started by myself in 2003 I registered the domain on September 16, 2003 (so technically it would be 11 years already :)). It started with a boot-disk, first created by hand and then based on yard. On 4th of October 2004 we had a first presentation of grml 0.09 Codename Bughunter at Kunstlabor in Graz. I managed to talk a good friend and fellow student Martin Hecher into joining me. Soon after Michael Gebetsroither and Andreas Gredler joined and throughout the upcoming years further team members (Nico Golde, Daniel K. Gebhart, Mario Lang, Gerfried Fuchs, Matthias Kopfermann, Wolfgang Scheicher, Julius Plenz, Tobias Klauser, Marcel Wichern, Alexander Wirt, Timo Boettcher, Ulrich Dangel, Frank Terbeck, Alexander Steinb ck, Christian Hofstaedtler) and contributors (Hermann Thomas, Andreas Krennmair, Sven Guckes, Jogi Hofm ller, Moritz Augsburger, ) joined our efforts. Back in those days most efforts went into hardware detection, loading and setting up the according drivers and configurations, packaging software and fighting bugs with lots of reboots (working on our custom /linuxrc for the initrd wasn t always fun). Throughout the years virtualization became more broadly available, which is especially great for most of the testing you need to do when working on your own (meta) distribution. Once upon a time udev became available and solved most of the hardware detection issues for us. Nowadays X.org doesn t even need a xorg.conf file anymore (at least by default). We have to acknowledge that Linux grew up over the years quite a bit (and I m wondering how we ll look back at the systemd discussions in a few years). By having Debian Developers within the team we managed to push quite some work of us back to Debian (the distribution Grml was and still is based on), years before the Debian Derivatives initiative appeared. We never stopped contributing to Debian though and we also still benefit from the Debian Derivatives initiative, like sharing issues and ideas on DebConf meetings. On 28th of May 2009 I myself became an official Debian Developer. Over the years we moved from private self-hosted infrastructure to company-sponsored systems, migrated from Subversion (brr) to Mercurial (2006) to Git (2008). Our Zsh-related work became widely known as grml-zshrc. jenkins.grml.org managed to become a continuous integration/deployment/delivery home e.g. for the dpkg, fai, initramfs-tools, screen and zsh Debian packages. The underlying software for creating Debian packages in a CI/CD way became its own project known as jenkins-debian-glue in August 2011. In 2006 I started grml-debootstrap, which grew into a reliable method for installing plain Debian (nowadays even supporting installation as VM, and one of my customers does tens of deployments per day with grml-debootstrap in a fully automated fashion). So one of the biggest achievements of Grml is from my point of view that it managed to grow several active and successful sub-projects under its umbrella. Nowadays the Grml team consists of 3 Debian Developers Alexander Wirt (formorer), Evgeni Golov (Zhenech) and myself. We couldn t talk Frank Terbeck (ft) into becoming a DM/DD (yet?), but he s an active part of our Grml team nonetheless and does a terrific job with maintaining grml-zshrc as well as helping out in Debian s Zsh packaging (and being a Zsh upstream committer at the same time makes all of that even better :)). My personal conclusion for 10 years of Grml? Back in the days when I was a student Grml was my main personal pet and hobby. Grml grew into an open source project which wasn t known just in Graz/Austria, but especially throughout the German system administration scene. Since 2008 I m working self-employed and mainly working on open source stuff, so I m kind of living a dream, which I didn t even have when I started with Grml in 2003. Nowadays with running my own business and having my own family it s getting harder for me to consider it still a hobby though, instead it s more integrated and part of my business which I personally consider both good and bad at the same time (for various reasons). Thanks so much to anyone of you, who was (and possibly still is) part of the Grml journey! Let s hope for another 10 successful years! Thanks to Max Amanshauser and Christian Hofstaedtler for reading drafts of this.

19 November 2014

Dirk Eddelbuettel: R / Finance 2015 Call for Papers

Earlier today, Josh send the text below to the R-SIG-Finance list, and I updated the R/Finance website, including its Call for Papers page, accordingly. We are once again very excited about our conference, thrilled about the four confirmed keynotes, and hope that many R / Finance users will not only join us in Chicago in May 2015 -- but also submit an exciting proposal. So read on below, and see you in Chicago in May! Call for Papers: R/Finance 2015: Applied Finance with R
May 29 and 30, 2015
University of Illinois at Chicago, IL, USA
The seventh annual R/Finance conference for applied finance using R will be held on May 29 and 30, 2015 in Chicago, IL, USA at the University of Illinois at Chicago. The conference will cover topics including portfolio management, time series analysis, advanced risk tools, high-performance computing, market microstructure, and econometrics. All will be discussed within the context of using R as a primary tool for financial risk management, portfolio construction, and trading. Over the past six years, R/Finance has included attendees from around the world. It has featured presentations from prominent academics and practitioners, and we anticipate another exciting line-up for 2015. This year will include invited keynote presentations by Emanuel Derman, Louis Marascio, Alexander McNeil, and Rishi Narang. We invite you to submit complete papers in pdf format for consideration. We will also consider one-page abstracts (in txt or pdf format) although more complete papers are preferred. We welcome submissions for both full talks and abbreviated "lightning talks." Both academic and practitioner proposals related to R are encouraged. All slides will be made publicly available at conference time. Presenters are strongly encouraged to provide working R code to accompany the slides. Data sets should also be made public for the purposes of reproducibility (though we realize this may be limited due to contracts with data vendors). Preference may be given to presenters who have released R packages. The conference will award two (or more) $1000 prizes for best papers. A submission must be a full paper to be eligible for a best paper award. Extended abstracts, even if a full paper is provided by conference time, are not eligible for a best paper award. Financial assistance for travel and accommodation may be available to presenters, however requests must be made at the time of submission. Assistance will be granted at the discretion of the conference committee. Please make your submission online at this link. The submission deadline is January 31, 2015. Submitters will be notified via email by February 28, 2015 of acceptance, presentation length, and financial assistance (if requested). Additional details will be announced via the R/Finance conference website as they become available. Information on previous years' presenters and their presentations are also at the conference website. For the program committee:
Gib Bassett, Peter Carl, Dirk Eddelbuettel, Brian Peterson, Dale Rosenthal,
Jeffrey Ryan, Joshua Ulrich

10 November 2014

Matthias Klumpp: Introducing Limba a software installer experiment

As some of you already know, since the larger restructuring in PackageKit for the 1.0 release, I am rethinking Listaller, the 3rd-party application installer for Linux systems, as well. During the past weeks, I was playing around with a lot of different ideas and code, to make installations of 3rd-party software easily possible on Linux, but also working together with the distribution package manager. I now have come up with an experimental project, which might achieve this. Motivation Many of you know Lennart s famous blogpost on how we put together Linux distributions. And he makes a lot of good and valid points there (in fact, I agree with his reasoning there). The proposed solution, however, is not something which I am very excited about, at least not for the use-case of installing a simple application[1]. Leaving things like the exclusive dependency on technology like Btrfs aside, the solution outlined by Lennart basically bypasses the distribution itself, instead of working together with it. This results in a duplication of installed libraries, making it harder to overview which versions of which software component are actually running on the system. There is also a risk for security holes due to libraries not being updated. The security issues are worked around by a superior sandbox, which still needs to be implemented (but will definitively come soon, maybe next year). I wanted to explore a different approach of managing 3rd-party applications on Linux systems, which allows sharing as much code as possible between applications. Limba Glick2 and Listaller concepts mergedlimba-small In order to allow easy creation of software packages, as well as the ability to share software between different 3rd-party applications, I took heavy inspiration from Alexander Larssons Glick2 project, combining it with ideas from the application-directory based Listaller. The result is Limba (named after Limba tree, not the voodoo spirit I needed some name starting with li to keep the prefix used in Listaller, and for a tool like this the name didn t really matter ;-) ). Limba uses OverlayFS to combine an application with its dependencies before running it, as well as mount namespaces and shared subtrees. Except for OverlayFS, which just landed in the kernel recently, all other kernel features needed by Limba are available for years now (and many distributions ship with OverlayFS on older kernels as well). How does it work? In order to to achieve separation of software, each software component is located in a separate container (= package). A software component can be an application, like Kate or GEdit, but also be a single shared library (openssl) or even a full runtime (KDE Frameworks 5 parts, GNOME 3). Each of these software components can be identified via AppStream metadata, which is just a few bits of XML. A Limba package can declare a dependency on any other software component. In case that software is available in the distribution s repositories, the version found there can be used. Otherwise, another Limba package providing the software is required. Limba packages can be provided from software repositories (e.g. provided by the distributor), or be nested in other packages. For example, imagine the software Kate requires a version of the Qt5 libraries, >= 5.2. The downloadable package for Kate can be bundled with that dependency, by including the Qt5 5.2 Limba package in the Kate package. In case another software is installed later, which also requires the same version of Qt, the already installed version will be used. Since the software components are located in separate directories under /opt/software, an application will not automatically find its dependencies, or be able to locate its own files. Therefore, each application has to be run by a special tool, which merges the directory trees of the main application and it s dependencies together using OverlayFS. This has the nice sideeffect that the main application could override files from its dependencies, if necessary. The tool also sets up a new mount namespace, so if the application is compiled with a certain prefix, it does not need to be relocatable to find its data files. At installation time, to achieve better system integration, certain files (like e.g. the .desktop file) are split out of the installed directory tree, so the newly installed application achieves almost full system integration. AQNAY* Can I use Limba now? Limba is an experiment. I like it very much, but it might happen that I find some issues with it and kill it off again. So, if you feel adventurous, you can compile the source code and use the example Foobar application to play around with Limba. Before it can be used in production (if at all), some more time is needed. I will publish documentation on how to test the project soon. Doesn t OverlayFS have a maximum stacking depth? Oh yes it has! The How does it work explanation doesn t tell the whole truth in that regard (mainly to keep the section small). In fact, Limba will generate a runtime for the newly installed software, which is a directory with links to the actual individual software components the runtime consists of. The runtime is identified by an UUID. This runtime is then mounted together with the respective applications using OverlayFS. This works pretty great, and also results in no dependency-resolution to be done immediately before an application is started. Than dependency stuff gives me a headache Admittedly, allowing dependencies adds a whole lot of complexity. Other approaches, like the one outlined by Lennart work around that (and there are good reasons for doing that as well). In my opinion, the dependency-sharing and de-duplication of software components, as well as the ability to use the components which are packaged by your Linux distribution is worth the extra effort. Can you give an overview of future plans for Limba? Sure, so here is the stuff which currently works: These features are planned for the new future: Remember that Limba is an experiment, still ;-) XKCD 927 Technically, I am replacing one solution with another one here, so the situation does not change at all ;-). But indeed, some duplicate work is done due to more people working in this area now on similar questions. But I think this is a good thing, because the solutions worked on are fundamentally different approaches, and by exploring multiple ways of doing things, we will come up with something great in the end. (XKCD reference) Doesn t the use of OverlayFS have an impact on the performance of software running with Limba? I ran some synthetic benchmarks and didn t notice any problems even the startup speed of Limba applications is only a few milliseconds slower than the startup of the raw native application. However, I will still have to run further tests to give a definitive answer on this. How do you solve ABI compatibility issues? This approach requires software to keep their ABI stable. But since software can have strict dependencies on a specific version of a software (although I d discourage that), even people who are worried about this issue can be happy. We are getting much better at tracking unwanted ABI breaks, and larger projects offer stable API/ABI during a major release cycle. For smaller dependencies, there are, as explained above, stricter dependencies. In summary, I don t think ABI incompatibilities will be a problem with this approach at least not more than they have been in general. (The libuild facilities from Listaller to minimize dependencies will still be present im Limba, of course) You are wrong because of $X! Please leave a comment in this case! I d love to discuss new ideas and find the limits of the Limba concept that s why I am writing C code afterall, since what looks great on paper might not work in reality or have issues one hasn t thought about before. So any input is welcomed! Conclusion Last but not least I want to thank Alexander Larsson for writing Glick2, which Limba is heavily inspired from, and for his patient replies to my emails. If Limba turns out to be a good idea, you can expect a few more blog posts about it soon.
* Answered questions nobody asked yet [1]: Don t get me wrong, I would like to have these ideas implemented they offer great value. But I think for simple software deployment, the solution is an overkill.

10 October 2014

Matthias Klumpp: Listaller + Glick: Some new ideas

As you might know, due to invasive changes in PackageKit, I am currently rewriting the 3rd-party application installer Listaller. Since I am not the only one looking at the 3rd-party app-installation issue (there is a larger effort going on at GNOME, based on Lennarts ideas), it makes sense to redesign some concepts of Listaller. Currently, dependencies and applications are installed into directories in /opt, and Listaller contains some logic to make applications find dependencies, and to talk to the package manager to install missing things. This has some drawbacks, like the need to install an application before using it, the need for applications to be relocatable, and application-installations being non-atomic. Glick2 There is/was another 3rd-party app installer approach on the GNOME side, by Alexander Larsson, called Glick2. Glick uses application bundles (do you remember Klik from back in the days?) mounted via FUSE. This allows some neat features, like atomic installations and software upgrades, no need for relocatable apps and no need to install the application. However, it also has disadvantages. Quoting the introduction document for Glick2: Bundling isn t perfect, there are some well known disadvantages. Increased disk footprint is one, although current storage space size makes this not such a big issues. Another problem is with security (or bugfix) updates in bundled libraries. With bundled libraries its much harder to upgrade a single library, as you need to find and upgrade each app that uses it. Better tooling and upgrader support can lessen the impact of this, but not completely eliminate it. This is what Listaller does better, since it was designed to do a large effort to avoid duplication of code. Also, currently Glick doesn t have support for updates and software-repositories, which Listaller had. Combining Listaller and Glick ideas So, why not combine the ideas of Listaller and Glick? In order to have Glick share resources, the system needs to know which shared resources are available. This is not possible if there is one huge Glick bundle containing all of the application s dependencies. So I modularized Glick bundles to contain just one software component, which is e.g. GTK+ or Qt, GStreamer or could even be a larger framework (e.g. GNOME 3.14 Platform ). These components are identified using AppStream XML metadata, which allows them to be installed from the distributor s software repositories as well, if that is wanted. If you now want to deploy your application, you first create a Glick bundle for it. Then, in a second step, you bundle your application bundle with it s dependencies in one larger tarball, which can also be GPG signed and can contain additional metadata. The resulting metabundle will look like this: glick-libundle This doesn t look like we share resources yet, right? The dependencies are still bundled with the application requiring them. The trick lies in the installation step: While the application above can be executed right away without installing it, there will also be an option to install it. For the user, this will mean that the application shows up in GNOME-Shell s overview or KDEs Plasma launcher, gets properly registered with mimetypes and is if installed for all users available system-wide. Technically, this will mean that the application s main bundle is extracted and moved to a special location on the file system, so are the dependency-bundles. If bundles already exist, they will not be installed again, and the new application will simply use the existing software. Since the bundles contain information about their dependencies, the system is able to determine which software is needed and which can simply be deleted from the installation directories. If the application is started now, the bundles are combined and mounted, so the application can see the libraries it depends on. Additionally, this concept allows secure updates of applications and shared resources. The bundle metadata contains an URL which points to a bundle repository. If new versions are released, the system s auto-updater can automatically pick these up and install them this means e.g. the Qt bundle will receive security updates, even if the developer who shipped it with his/her app didn t think of updating it. Conclusion So far, no productive code exists for this I just have a proof-of-concept here. But I pretty much like the idea, and I am thinking about going further in that direction, since it allows deploying applications on the Linux desktop as well as deploying software on servers in a way which plays nice with the native package manager, and which does not duplicate much code (less risk of having not-updated libraries with security flaws around). However, there might be issues I haven t thought about yet. Also, it makes sense to look at GNOME to see how the whole 3rd-party app deployment issue develops. In case I go further with Listaller-NEXT, it is highly likely that it will make use of the ideas sketched above (comments and feedback are more than welcome!).

11 September 2014

Sylvestre Ledru: Rebuild of Debian using Clang 3.5.0

Clang 3.5.0 has just been released. A new rebuild has been done highlight the progress to get Debian built with clang. tl;dr: Great progress. We decreased from 9.5% to 5.7% of failures. Full results are available on http://clang.debian.net At time of the rebuild with 3.4.2, we had 2040 packages failing to build with clang. With 3.5.0, this dropped to 1261 packages. Fixes With Arthur Marble and Alexander Ovchinnikov, both GSoC students, we worked on various ways to decrease the number of errors. Upstream fixes First, the most obvious way, we fixed programming bugs/mistakes in upstream sources. Basically, we took categories of failure and fixed issues one after the other. We started with simple bugs like 'Wrong main declaration', 'non-void function should return a value' or 'Void function should not return a value'.

They are trivial to fix. We continued with harder fixes like ' Undefined reference' or 'Variable length array for a non POD (plain old data) element'.

So, besides these one, we worked on:
In total, we reported 295 bugs with patches. 85 of them have been fixed (meaning that the Debian maintainer uploaded a new version with the fix).

In parallel, I think that the switch by FreeBSD and Mac OS X to Clang also helped to fix various issues by upstreams. Hacking in clang As a parallel approach, we started to implement a suggestion from Linus Torvalds and a few others. Instead of trying to fix all upstream, where we can, we tried to update clang to improve the gcc compatibility.

gcc has many flags to disable or enable optimizations. Some of them are legacy, others have no sense in clang, etc. Instead of failing in clang with an error, we create a new category of warnings (showing optimization flag '%0' is not supported) and moved all relevant flags into it. Some examples, r212805, r213365, r214906 or r214907

We also updated clang to silent some useless arguments like -finput_charset=UTF-8 (r212110), clang being UTF-8 compliant.

Finally, we worked on the forwarding of linker flags. Clang and gcc have a very different behavior: when gcc does not know an argument, it is going to forward the argument to the linker. Clang, in this case, is going to reject the argument and fail with an error. In clang, we have to explicitly declare which arguments are going to be transfer to the linker. Of course, the correct way to pass arguments to the linker is to use -Xlinker or -Wl but the Debian rebuild proved that these shortcuts are used. Two of these arguments are now forwarded: New errors Just like in other releases, new warnings are added in clang. With (bad) usage of -Werror by upstream software, this causes new build failures: I also took the opportunity to add some further categorizations in the list of errors. Some examples: Next steps The Debile project being close to ready with Cl ment Schreiner's GSoC, we will now have an automatic and transparent way to rebuild packages using clang. Conclusion As stated, we can see a huge drop in term of number of failures over time:
Hopefully, Clang getting better and better, more and more projects adopting it as the default compiler or as a base for plugin/extension developments, this percentage will continue to decrease.
Having some kind of release goal with clang for Jessie+1 can now be considered as potentially reachable. Want to help? There are several things which can be done to help: Acknowledgments Thanks to David Suarez for the rebuilds of the archive, Arthur Marble and Alexander Ovchinnikov for their GSoC works and Nicolas S velin-Radiguet for the few fixes.
Original post blogged on b2evolution.

Next.

Previous.